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ABSTRACT 


Facial emoji recognition is a human-computer interaction system. In recent 
times, automatic face recognition or facial expression recognition has attracted 
increasing attention from researchers in psychology, computer science, 
linguistics, neuroscience, and similar fields. Facial emoji recognizer is an end 
user application which detects the expression of the person in the video being 
captured by the camera. The smiley relevant to the expression of the person in 
the video is shown on the screen which changes with the change in the 
expressions. Facial expressions are important in human communication and 
interactions. Also, they are used as an important tool in studies about behavior 
and in medical fields. Facial emoji recognizer provides a fast and practical 
approach for non-meddlesome emotion detection. The purpose was to develop 
an intelligent system for facial based expression classification using CNN 
algorithm. Haar classifier is used for face detection and CNN algorithm is utilized 
for the expression detection and giving the emoticon relevant to the expression 
as the output. 
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1. INTRODUCTION 

Facial emotion recognition and analysis has been gaining a 
great attention in the advancement of human machine 
interface as it provides a natural and efficient way to 
communicate between humans. Some application are as 
related to face and its expressions include person 
identification and access control, video call and 
teleconferencing, forensic applications, human-computer 
interaction, automated surveillance, cosmetology, and so on. 
But the performance of the face expression detection 
certainly affects the performance of all the applications. 



Many methods have been suggested to detect human face in 
pictures and videos, they can be divided into four types: 
knowledge-based methods, feature-based methods, template 
based methods and appearance-based methods. When these 
methods are used separately, they cannot solve all the 
problems of face detection like pose, expression, orientation. 
Hence it is recommended to operate with several successive 
or parallel methods. Most of the existing facial expression 
recognition methods which are popularly used till today are 
focused on recognition of five primary expression categories 
such as: happiness, sadness, fear, anger and disgust. We 
implement HAAR classifier for face detection, CNN algorithm 
for expression detection and two functions are used: relu 
and soft max(these are activity functions]. 


Facial expressions play a major role in enhancing 
communication and interactions between people. Also, they 
are used as an important tool tool in psychological studies 
and in medical rehabilitation. 

Face based emotion detection techniques may give a rapid 
and feasible approach for non-invasive emotion detection, to 
investigate emotional recognition using facial expression by 
emoji in real time. 

The purpose is to develop an feasible system for facial image 
or video based expression detector. The objective of this 
project is to understand the facial emotion recognition in 
real time and develop Automatic Facial Expression 
Recognition System which can take video as input and 
recognize and classify it into five different expression. 
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2. RELATED WORK 

Darwin was the first to imply that some of these facial 
expressions of emotion have their origins in the evolution of 
the human species. 

These face expressions helped the living things live because 
it looked necessary to social animals like humans or 
chimpanzees to express these forthcoming behaviors given 
by the emotions so they could avoid fights, conflict, 
disagreement, danger, or allow comfort, approach, and so 
forth. However, these emotion expressions are not capable of 
modification by social learning. 

Different cultures apply different display rules to direct their 
expressions of emotion. Although the current evidence 
supports Darwin's basic premise it is not without 
controversy. Future technological advances will allow facial 
expression research to expand to address many of the 
important issues that remain. 

Ekman and Friesen observed the facial muscles which are 
important in expressing emotion and made their findings to 
a system of 46 action units (AUs). These action units, in 
which some are raising of the inner eyebrow and raising of 
the outer eyebrow, were extremely important in considering 
human expressions. Before this system was published, facial 
expression research relied heavily on human labelling of 
example expressions and many researchers were concerned 
about bias related to cultural context or the labeller’s 
emotional state at the time. The advent of Ekman and 
Feisen’s facial action coding system in 1977 put these 
concerns to rest and quickly became the golden standard. 

In 2004, Paul Viola and Michael Jones developed an 
extremely efficient face detector with high performance by 
using an adaboost learning algorithm to classify features 
derived from Haar-like features. This method was applied by 
Wang et al for facial expressions he later was able to sort 
faces into one of 7 archetypal facial expression with 92.4% 
accuracy. An investigation by White hill and Omlin found 
that the use of Haar features in combination with the Ada 
boost boosting algorithm was at least two orders of 
magnitude faster than the more standard classification using 
SVMs and Gabor filters without a significant drop in 
performance. 

More recently, work done by Happy et al in 2015 found that 
the viola-jones algorithm was valuable not in direct 
detection of emotions but as a pre-processing step to 
efficiently detect the most important face regions (IE lip 
corners and eyebrow edges) which where then further 
processed with local binary pattern histograms. Their 
solution performed similarly to other state-of- the-art 
methods but required far less computational time. 

Eigen faces are computed for all the training images and 
classification of test image is done by the following steps: 
Generate vector of all the images and matrix of vector is 
created for each image type, 

Mean of the training faces is created, Subtract the test image 
with each of the mean image, Co-variance matrix is 
calculated for each of the difference vectors created. 

3. PROBLEM STATEMENT 

Facial emoji recognizer is an end user application which 
detects the expression of the person in the video being 


captured by the camera. The smiley relevant to the 
expression of the person in the video is shown on the screen 
which changes with the change in the expressions. We 
implement HAAR classifier for face detection, CNN algorithm 
for expression detection and two functions are used: relu 
and soft max(these are activity functions) 


4. METHODOLOGY 



Facial expressions can be described as the arrangement of 
facial muscles to convey a certain emotional state to the 
observer in simple words. Emotions can be divided into six 
broad categories—Anger, Disgust, Fear, Happiness, Sadness, 
Surprise, and Neutral. In this, train a model to differentiate 
between these, train a convolutional neural network using 
the FER2013 dataset and will use various hyper-parameters 
to fine-tune the model. 

1. Decomposing an image. 

Images are composed of pixels and these pixels are nothing 
more than numbers. Often it is considered that the Coloured 
images can be divided into three colour channels, which are 
:red, green, and blue and each channel is represented by a 
grid (2-dimensional array). Each cell in the grid stores a 
number between 0 and 255 which denotes the intensity of 
that cell. 

2. Importing Necessary Libraries 

3. Define Data Loading Mechanism 

Now, we will define the load_data() function which will 
efficiently parse the data file and extract necessary data and 
then convert it into a usable image format. 

All the images in our dataset are 48x48 in dimensions. Since 
these images are gray-scale, there is only one channel. We 
will extract the image data and rearrange it into a 48x48 
array. Then convert it into unsigned integers and divide it by 
255 to normalize the data. 255 is the maximum possible 
value of a single cell and by dividing every element by 255, 
we ensure that all our values range between 0 and 1. 
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We will check the Usage column and store the data in 
separate lists, one for training the network and the other for 
testing it. 

4. Defining the model. 

We will use Keras to create a Sequential Convolutional 
Network. Which means that our neural network will be a 
linear stack of layers. This network will have the following 
components: 

A. Convolutional Layers: These layers are the building 
blocks of our network and these compute dot product 
between their weights and the small regions to which 
they are linked to. This is considered as the method in 
which layers learn certain features from these images. 

B. Activation functions: are those functions which are 
applied to the outputs of all layers in the network. In this 
project, we will resort to the use of two functions— R elu 
and Soft max. 

C. Pooling Layers: These layers will down sample the 
operation along the dimensions. This helps reduce the 
spatial data and minimize the processing power that is 
required. 

D. Dense layers: These layers are present at the end of a 
C.N.N. and these take in all the feature data generated by 
the convolution layers and does the decision making. 

E. Dropout Layers: randomly turns off few neurons in the 
network to prevent over fitting. 

5. Callback functions 

Callback functions are those functions which are called after 
every epoch during the training process. We will be using the 
following callback functions: 

1. Reduce LR On Plateau: Training a neural network can 
plateau at times and we stop seeing any progress during 
this stage. Therefore, this function monitors the 
validation loss for signs of a plateau and then alter the 
learning rate by the specified factor if a plateau is 
detected. 

lr_reducer = Reduce LR On 

Plateau(monitor='val_loss', factor=0.9, patience=3) 

2. Early Stopping: At times, the progress stalls while 
training a neural network and we stop seeing any 
improvement in the validation accuracy [in this case]. 
Majority of the time, this means that the network won’t 
converge any further and there is no point in continuing 
the training process. This function waits for a specified 
number of epochs and terminates the training if no 
change in the parameter is found. 

early_stopper= 

EarlyStopping(monitor='val_acc',in_delta=0, 
patience=6, mode='auto') 

Model Checkpoint: Training neural networks generally take 
a lot of time and anything can happen during this period that 
may result in loss of all the variables and weights. 

Creating checkpoints is a good habit as it saves your model 
after every epoch. In case your training stops you can load 
the checkpoint and resume the process. 


6. Train 

7. Test the model 

The project is started off by defining a loading mechanism 
and loading the images. Then a training set and a testing set 
are created. After this a fine model and a few callback 
functions are defined. The basic components of a 
convolutional neural network are considered and then we 
training is done to the network. 


5. OUTPUT 



In the above figure, i.e., Sample output is the window where 
the user expressions are captured by the web cam and the 
respective emotions are detected. 

On detecting the emotion, respective emoticon is shown on 
the left side of the screen. This emoticon changes with the 
change in the expression of the person in front of the web 
cam. Hence changes with the change in the expression of the 
person in front of the web cam. Hence this real time 
application is very beneficial in various fields like 
psychology, computer science linguistics,neuroscience and 
related disciples. 

6. APPLICATIONS 

Robust spontaneous expression recognizers can be 
developed and deployed in real-time systems and used in 
building emotion sensitive HC1 interfaces. The project can 
have an impact on our day to day life by enhancing the way 
we interact with computers or in general, our surrounding 
living and work spaces. High correct recognition rate , 
significant performance improvements in our system. 
Promising results are obtained under face registration 
errors, fast processing time. 

System is fully automatic and has the capability to work with 
video feeds as well as images. It is able to recognize 
spontaneous expressions. Our system can be used in Digital 
Cameras where in the image is captured only when the 
person smiles, or if the person doesn’t blink his eyes. In 
security systems which can identify a person, in any form of 
expression he presents himself. Rooms in homes can set the 
lights, television to a persons taste when they enter the 
room. 

This can be used by the doctors in understanding the 
intensity of pain or illness of a deaf patient. 

7. CONCLUSION 

Proposed is a human emotion detector using emoticon using 
machine learning, python to predict emotions of the people 
and represent them using emoticon.. These include image 
acquisition, preprocessing of an image, face detection, 
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feature extraction, classification and then when the emotions 
are classified the system assigns the user particular music 
according to his emotion. Our system focuses on live videos 
taken from the webcam. 

The main aim of this project is to develop automatic facial 
emotion recognition system in which an emoticon is used for 
giving the output for individuals thus assigning them various 
therapies or solutions to relief them from stress. The 
emotions used for the experiments include happiness, 
Sadness, Surprise, Fear, Disgust, and Anger that are 
universally accepted. 
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